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ABSTRACT 


High school GPAs (HSGPAs) are often perceived to represent inconsistent levels of readiness 
for college across high schools, while test scores (e.g., ACT scores) are seen as comparable. This 
study tests those assumptions, examining variation across high schools of both HSGPAs and 
ACT scores as measures of academic readiness for college. We find students with the same 
HSGPA or the same ACT score graduate at very different rates based on which high school they 
attended. Yet, the relationship of HSGPAs with college graduation is strong and consistent, and 
larger than school effects. In contrast, the relationship of ACT scores with college graduation is 
weak, smaller than high school effects, and the slope of the relationship varies by high school. 
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High school course grades are critical indicators of academic performance for students, 
educators, and institutions of higher education. Yet, standardized test scores are often seen as 
more reliable and objective indicators of academic preparation than students’ grades because all 
students are judged based on the same tasks under the same conditions. All states use 
standardized tests to judge students’ progress toward college readiness goals, with 45 states using 
ACT or SAT scores (Nayar, 2015). The use of standardized test scores to monitor students’ 
college readiness is recommended clearly in the What Works Clearinghouse Practice Guide on 
how to prepare students for college, while HSGPAs are discussed as one piece of performance 
data to consider, along with curriculum and assessments (Tierney, Bailey, Constantine, 
Finkelstein, & Hurd, 2009). A key assumption behind the emphasis on test scores in policy and 
practice is that college entrance exams are strong and consistent measures of readiness. Yet, the 
emphasis on test scores over grades in policy and practice recommendations stands in contrast to 
research showing high school grade point averages (HSGPAs) are stronger predictors than test 
scores of college outcomes (Bowen, Chingos, & McPherson, 2009; Geiser & Santelices, 2007; 
Hiss & Franks, 2014; Kobrin, Patterson, Shaw, Mattern, & Barbuti, 2008). 

In this study, we directly address questions about the variability in HSGPAs across high 
schools as predictors of college readiness, examining whether students with the same HSGPAs 
are systematically more likely to graduate college if they came from particular high schools, and 
whether the slope of the relationship differs by high school. We then conduct the same tests with 
ACT scores. We also discern the extent to which there are high school effects on college 


graduation that are not captured in either students’ HSGPAs or ACT scores. 
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Prior Literature on the Reliability of Course Grades across Schools and Validity of Tests 


Numerous publications give the impression that course grades are not reliable measures 
of achievement in comparison with test scores. For example, the introduction of a new book on 
testing and college admissions states: 

...standardized admissions tests provide a neutral yardstick to assess the 
performance and promise of students from secondary schools whose course 

offerings differ sidely in variety and rigor. This is a particularly salient point in an 

era of widespread grade inflation ... (Buckley, Letukas, & Wildavsky, 2018). 

Likewise, the introduction of a new report by the Fordham Foundation expresses 
concern that teachers’ grades do not reflect state standards, and wonders how to help 
parents put more faith in test scores as measures of their students’ readiness instead of 
relying so much on grades (Northern & Petrilli, 2018). These documents reflect current 
beliefs, which are echoed in the emphasis placed on test scores in policy and in practice 
recommendations, described above, and often inferred without strong evidence in 


research studies. However, the evidence is not strong for these beliefs, as described 


below. 


Grades can be seen as non-comparable across schools because they are based on 
criteria developed by individual teachers, in schools with different curricula. Grades are 
assigned based on a potentially wide-ranging array of tasks, measured over time, capturing 
academic knowledge, skills, and academic behaviors, effort, and incorporating teacher 
judgement. (Bowers, 2011; Brookhart, 1993; Brookhart et al., 2016; Farkas, Sheehan, Grobe, & 
Shuan, 1990; Kelly, 2008). The fact that grades are based on a wide range of factors, with 


judgement from many different teachers, makes them potentially highly variable across contexts. 


Are GPAs an Inconsistent Measure across High Schools? 


At the same time, the fact that they are based on a large number of raters (teachers) across a wide 
range of relevant tasks, could actually make them very reliable as indicators of academic 
readiness for college, where students will also be asked to do a wide range of tasks with different 
expectations, assessed by many different instructors. 

There is no reason to believe a priori that tests would necessarily be more reliable than 
grades as predictors of college performance. Standardized tests assess students on a narrow range 
of skills (mostly a subset of what students learn in English and math classes) in one type of 
condition (a timed test), while colleges expect students to have broad knowledge and skills 
across many subjects, and to show consistent effort in different types of assignments over 
months at a time. Schools could prepare students for the tests in very different ways (see Koretz, 
2017), with different implications for their students’ readiness for college. 

Moderate correlations with test scores are often used as evidence of unreliability in 
grades. People sometimes make the argument that grades are “inflated” or “subjective” based on 
evidence that HSGPAs have increased over time, without concurrent changes in test scores 
(Camara, Kimmel, Scheuneman, Sawtell, 2004; Gershenson, 2018; Godfrey, 2011; Hurwitz & 
Lee, 2018), or that students with the same test scores have different HSGPAs at different schools 
(U.S. Department of Education, 1994; Woodruff & Ziomek, 2004). Pattison, Grodsky, and 
Muller (2013) describe some of the conceptual flaws in the argument that grades should align 
tightly with standardized test scores, and suggest focusing instead on the predictive validity of 
each for later outcomes. 

Evidence about the validity of standardized test scores as measures of college readiness 
has its own weaknesses, making it questionable to use standardized tests as a metric for judging 


the reliability of grades. SAT and ACT validitation studies tend to be based on improvement in 
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the prediction of college freshman GPAs when test scores are used together with student- 
reported HSGPA, relative to models that use student-reported HSGPA alone (e.g., Kobrin, et al., 
2008; Noble & Sawyer, 2002; Woodruff & Ziomek, 2004). Researchers argue that because the 
test scores improve the prediction of college freshman GPAs over and above student-reported 
HSGPAs, they are valid indicators to adjust for inconsistencies in HSGPAs. However, student- 
reported HSGPAs are more weakly correlated with college freshman GPAs than unweighted 
HSGPAs taken from transcripts (Geiser & Santelices, 2006; Kuncel, Credé, & Thomas, 2005; 
Zwick & Himelfarb, 2011); the studies likely over-estimate the value that test scores provide. 

Studies based more heavily on HSGPAs from transcripts than student reports suggest test 
scores provide little improvement in the prediction of college outcomes. Using data from a large 
sample of colleges across the country, Bowen, Chingos, and McPherson (2009) found the 
relationship of SAT and ACT scores with college outcomes was small and sometimes not 
significant (depending on institution type), controlling for HSGPAs, comparing students in the 
same colleges. In contrast, HSGPAs had a strong relationship with college outcomes controlling 
for students’ test scores. Hiss and Franks (2014) concluded that students in test-optional colleges 
who did not submit test scores had similar or better college outcomes than students in the same 
colleges with similar HSGPAs who did submit scores, even though their scores on standardized 
tests were much lower. Using data from California universities, Rothstein (2004) found that most 
of the relationship of SAT scores with college GPA could be attributed to high school poverty, 
school racial composition, and student background. 

Grades are lower in harder classes with stronger peers, and this suggests 
inconsistency in HSGPAs. A number of studies have discerned what are called “frogpond” 


effects (Attewell, 2001), where students with similar prior test scores, academic performance, or 
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effort receive lower grades in classrooms and schools of predominantly high-achieving students, 
as compared to those with lower-achieving students (Farkas et al., 1990; Nomi & Allensworth, 
2009; Barrow, Sartain, & de la Torre, 2016). Students also tend to get lower grades in classes 
that are intentionally designed to be challenging, such as Advanced Placement and Honors 
courses (Sadler & Tai, 2007). 

Differences in the types of classes that students take, and the expectations associated with 
the peer composition, introduce “noise” into the metric of HSGPAs as an indicator of academic 
performance in high school. ACT and SAT validity studies claim that students’ test scores can be 
used to adjust for different standards and expectations at different schools. There is a need to 
evaluate that claim using data on HSGPAs from transcripts. It is possible that the overall 
achievement level in a student’s school—information that is publicly available—might be just as 
useful, or more useful, than individual student’s test scores. Two prior studies note that the size 
of the relationship between HSGPA and college outcomes (graduation or college GPAs) is larger 
among students within the same high school (i.e., when high school fixed effects are used in a 
model), than across schools (Bowen et al., 2009; Koretz & Langi, 2018). They interpret this as 
meaning that HSGPA represents a higher level of achievement at some schools than others, 
which would be consistent with the “frogpond” effects discussed above, and suggest adjusting 
for these differences with information on school average achievement. 

Different college outcomes for student subgroups with the same HSGPAs have been 
used as evidence of different grading standards. Other studies have suggested that HSGPAs 
are inconsistent measures of achievement across high schools because HSGPAs predict that 
Black and Latino students, and students from low-SES high schools, will receive higher college 


grades than they actually do (Zwick & Himelfarb, 2011; Zwick, 2013). The researchers suggest 
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that this discrepancy results from differences in the quality of high schools attended by students, 
and show that school average poverty, used as a proxy for school quality, accounts for some of 
the differences. They do not conduct equivalent tests of SAT scores to see if similar or larger 
discrepancies by race or school poverty would occur with prediction models that use SAT scores 
alone. They also find discrepancies to be much smaller when they use HSGPAs from transcripts, 
rather than student-reported HSGPAs. Their arguments hold if one is to believe that race, 
ethnicity, and SES do not affect college success in ways unrelated to acacemic readiness. There 
are many reasons to believe this is not true, and studies have shown that SAT and ACT scores 
also overpredict college performance for the same groups (Noble, 2004; Bridgeman, McCamley- 
Jenkins, & Ervin, 2000; Rothstein, 2004). However, this does suggest that it is important to 
compare students with similar backgrounds when evaluating the validity of HSGPAs or test 
scores as indicators of college readiness, and suggests there may be high school effects on 
students’ college outcomes that need to be better understood. 

High schools could effect college outcomes in many ways that are not reflected in 
either students’ HSGPA or test performance. For example, high schools might develop 
structures to prepare students with more “college knowledge” to navigate the post-secondary 
realm (Hoxby & Turner, 2015; Conley, 2008), or provide a more diverse environment that 
teaches students to adjust to new situations and people (Tam & Bassett, 2004). Fletcher and 
Tienda (2007) found that high school fixed effects explained half or more of the differences in 
college GPA and graduation by students’ race and ethnicty, sometimes reversing the 
relationships. Such high school effects could make it appear that HSGPAs have different value in 
some schools than others—when there are simply other factors about high schools that also 


matter for college success. 
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Contributions of this Study and Research Questions 


In this study, we compare the strength and consistency of HSGPAs as predictors of 


college graduation across high schools with ACT scores, adding to the current literature in a 


number of ways: 


1) 


2) 


3) 


Showing variation across high schools in the relationship of HSGPA with college 
graduation. Variation in the predictiveness of HSGPAs by high school could occur 
either because HSGPAs represent higher levels of readiness from some high schools vs. 
others (i.e., HSGPAs under- or over-predict graduation for all students at a school), or 
because the relationship (slope) of the HSGPA differs across high schools (1.e., providing 
a stronger signal at some schools than others). Prior research has not shown the extent to 
which there is variation in the relationship of HSGPAs with college graduation by high 
school. 

Conducting equivalent tests on ACT scores as on HSGPA. Past studies have not 
explicitly tested whether standardized assessments are comparable across high schools as 
measures of college readiness. We examine whether students with the same ACT or SAT 
scores have different college outcomes based on which high school they attended, or if 
the slope of the relationship of test scores to college outcomes varies by high school. 
Discerning the extent to which there are high school effects on college graduation 
that are not captured in either students’ HSGPAs or ACT scores. While past studies 
have provided evidence that high school effects on college outcomes exist, they have not 


quantified the magnitude of high school effects (e.g., the variance across high schools), 
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that is, how much of a difference it makes which high school a student attended for 


students who look similar based on their ACT score and HSGPA. 


We begin by identifying the extent to which the relationship of each achievement 
indicator depends on the high school a student attends: 


RQ1: How different are college graduation rates for students with the same 
HSGPAs/ACT scores, who come from different high schools? 


We then compare the size and consistency of the relationships of HSGPAs and ACT 
scores with college graduation, and examine whether including students’ ACT scores in the 
prediction of college graduation substantially reduces inconsistency across high schools over 


using HSGPA alone: 


RQ2: Do ACT scores provide a stronger, or more consistent, prediction of college 
readiness across high schools than HSGPAs? 


RQ3: Is there less high school variance in college graduation rates in models that use 
students’ ACT scores and HSGPAs together, than models that use HSGPAs alone? 


Finally, we show the extent to which information about high schools (school poverty and 


average ACT scores) explain high-school level variation. 


RQ4: To what extent are high school differences in college graduation rates for students 
with the same HSGPAs and ACT scores explained by school achievement level and 
school poverty? 


Research Methods 


This study uses data from the Chicago Public Schools (CPS), a large, public school 


district that contains schools with varying academic composition—extremely high-achieving 
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selective schools that get ranked among the top high schools in the country, heterogeneous 
schools, and schools with very low test scores. We include for analysis all students who 
graduated from neighborhood, magnet, selective, and vocational high schools between the years 
of 2006 and 2009, who enrolled in a four-year college immediately following graduation, and 
who had complete data (n=17,753).' Table 1 provides summary statistics of the analytic group 
and variables used in the models. 

We only include students who enrolled in a four-year college, so as not to confound 
enrollment in college with ability to succeed in college once enrolled. Because college 
admissions use HSGPA and ACT scores to determine who is accepted, those measures will be 
related to college graduation simply because they provide access, regardless of whether they 
indicate readiness to succeed once enrolled. By comparing only students who enrolled in college, 
and controlling for institutional characteristics (described below), we focus on the extent to 
which the HSGPAs and ACT scores are indicators of students’ likelihood of succeeding once in 


college, not the degree to which they are signals to admissions officers. 


Data and Variables 


Data on academic performance and student demographic information (gender, race, and 
ethnicity) come from district administrative datasets. We obtained economic information on 
students’ residential neighborhoods by linking students’ addresses to information from the U.S. 
Census at the level of block groups on the percent of adult males employed and the percent of 


families with incomes above the poverty line. HSGPAs were created by coding grades in 


' Charter school graduates were not included because their transcripts are not available. A total of 2,595 cases had 
missing data: 934 were missing cumulative HSGPAs, 982 were missing ACT scores, and 828 were missing the 
IPEDS institutional graduation rate for the college they attended. Students in the restricted group had slightly higher 
HSGPAs (2.72 vs 2.69) and ACT scores (20.12 vs 19.97) than students in the total population. The groups were 
nearly identical with regards to ethnicity, race, gender, SES, and institutional graduation rate. 
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students’ transcripts 0 through 4 (F through A), and creating an unweighted average of all 
courses completed in high school. At the time the students were in high school, all students in 
Illinois took the ACT during the spring of the eleventh grade. College enrollment records and 
six-year graduation outcomes were obtained through the National Student Clearinghouse. 
Students were included for analysis if they had full-time enrollment records in a four-year 
college during the fall term after they graduated high school. Six-year college graduation is 
defined as earning a four-year college degree within six years of graduating from high school. 
Appendix Table Al shows college graduation rates by students’ ACT scores and HSGPA 
unadjusted for demographic and college characteristics or high school effects. 

Colleges offer different supports and structures which influence whether students 
graduate (Bowenet al., 2009; Cohodes & Goodman, 2012; Kurlaender & Grodsky, 2013). 
Therefore, it was important to control for college characteristics. We did this by including 
information on colleges obtained through the Integrated Postsecondary Education Data System 
(IPEDS) as covariates: the race- or ethnicity-specific six-year institutional graduation rate of the 
college (matched to the race and ethnicity of the student), college size (number of undergraduate 


students), the percentage of freshman students who are full time, and the student-to-faculty ratio. 


Methods 


We estimated the variance in college graduation rates by high school using hierarchical 
linear models, with students nested within high schools. We considered using cross-nested 
models with students simultaneously nested within their high school and college, or controlling 
for college fixed effects. However, students in our analysis group matriculated to more than 500 


different four-year colleges across the U.S., and at many of these colleges there were only a 
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small number of students. This resulted in imprecise estimates of college effects for a large 
proportion of the sample through these other methods. 

For RQI, we used two different methods of estimating variation in college graduation 
rates by high school for students with the same HSGPA/ACT score. First, we used a series of 
dummy variables to model the relationship between HSGPA and college graduation non- 
parametrically. There is no intercept, so the coefficient for each HSGPA dummy variable 
represents the average college graduation rate for students in that HSGPA group. We allowed the 
coefficients to vary by high school to identify the variation in college graduation rates across 
high schools for students with the same HSGPAs. These same models were then repeated with 
student ACT bins in lieu of HSGPA bins. Coefficients for other covariates were fixed across 
schools, predicting the log odds of graduating from a four-year college in six years: 

Level-1 Model [1] 


log (Peraa/1-Pgraa)ix= L8=1 Boj (S)ij + yer Bai (G)ij + e222 Boj (C)ijt tj 
Level-2 Model 


Bsj = Yso 
Bei = Yeo + Ugj 
Boj = Yeo 


S is a vector of student background variables (Neighborhood poverty, male, 
Black, Latino, and Asian). 


G is a vector of dummy variables representing HSGPA bands. 

C is a vector of college institutional variables. 

Ugj is the high school-level variance in college graduation rates for students in the 
HSGPA band, controlling for student background and college institutional 
variables. 

The above method assumes no particular functional form. However, because students 


with different levels of achievement are not evenly distributed across schools, not all high 


schools have students in all achievement bands. Therefore, we only calculated school-level 
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random effects for bands in which at least 95 percent of schools are represented. 

We also ran models which use standardized continuous versions of HSGPAs, rather than 
the binned variables, and calculated the average school effect across all achievement levels. 
These models include a squared term, since the relationship of each achievement measure is 
slightly quadratic. We ran models in which the slopes of each achievement measure with college 
graduation are fixed, and models that allow the slopes of the relationships to vary by high school. 
As the results are similar, only the second are shown in the manuscript, and the first are available 
from the authors: 


Level-1 Model [2] 
log (Perad/1-Pgraa) i= Boj + Dey Boj (S)ij + Bej(ZGPA)ij+ B7j(ZGPA*);; + 
en Boj (C)ijt ij 


Level-2 Model 
Boj = Yoo + Uo; 
Bs = Ys0 
Boj = Yoo + U6j 
Brj = y70 + U7 
Bej = Yeo 
In Equation 2, uo; is the high school-level variance in college graduation rates, controlling 
for students’ HSGPA, student background variables and the institutional characteristics of the 
colleges in which they enroll. Variance components on the slopes, U6; and u7;, show variation in 
the size the relationship of HSGPA with college graduation across high schools--whether grades 
are stronger measures of college readiness at some schools than others. Equations 2 was 
replicated with ACT scores. 
Finally, we ran models that entered HSGPA and ACT scores together in the models to 


discern how much ACT scores improve the prediction of college graduation beyond using 


HSGPAs alone, to answer question 3: 
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Level-1 Model, Equation 2 [3] 
log (Pgraa/1-Pgraa) j= Boj + Yet Boi (S)ij + Bej(ZGPA)ij+ B7j(ZACT) ji; + Bgj(ZGPA?);; + 
Boj (ZACT?) iz + De310 Boj (C)ijt Vij 
Level-2 Models 
Boj = Yoo + Uo; 
Bsj = Yso 
Boj = Yoo + Ugj 
B7j = y70 + U7) 
Boj = Ys0 + gi 
B7j = Yoo + Us; 


Bej = Vc0 


We did this in two ways. First, we grand-mean centered all student variables to show the 
overall relationships, and then we group-mean centered the variables to discern the relationship 
of each with college graduation relative only to other students in the same school. This second 
specification is similar to a school fixed-effects model. To address RQ4, we included school- 
level predictors of school performance level (average ACT score) and school poverty as 


predictors of Boj. 


Results 

Table 2 displays coefficients from models predicting college graduation rates with 
HSGPA, without and with covariates. The odds ratios show the likelihood of graduating from 
college; students with a 3.0-3.25 HSGPA have fairly even odds (0.91), which gives them just 
under a 50-50 chance (48 percent probability), while students with a HSGPA of 3.5-3.75 are 3.6 
times more likely to graduate as to not graduate (odds of 3.65, or about 78 percent graduating 
and 22 percent not graduating). HSGPA has a strong relationship with college graduation in both 
the unconditional model and the model that controls for students’ backgrounds and college 


institutional variables, although the relationship is smaller once the control variables are 
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introduced. The coefficients from the full model are converted into percentages and displayed 
graphically as the thick black line in the left panel of Figure 1. Across the range of HSGPAs, the 
probability of graduating from college ranges from 20 percent for students with HSGPAs less 
than 1.5 to about 80 percent for students with HSGPAs of 3.75 or higher, after controlling for 
student backgrounds and college characteristics. 

The random effects at the bottom of Table 2 show the degree to which average 
graduation rates vary across high schools among students in each HSGPA bin. There is 
significant high school variance in college graduation rates for students in each HSGPA bin. For 
example, among students with HSGPAs between 3.25-3.5, a two-standard deviation range of 
high school effects is 0.144 + 0.575 in log-odds in the conditional model. Thus, students with a 
3.25-3.5 HSGPA at schools with very negative school effects (one standard deviation below the 
mean) have college graduation rates that are similar to students with HSGPAs of 2.75-3.0 at 
more typical schools (where the odds of graduating are 0.72). 

Model 2 in Table 2 shows the results from a model where HSGPA is entered as a 
continuous variable along with a squared term, instead of discrete bins. The linear component 
shows that for every standard deviation increase in HSGPA, the odds of graduating from college 
double (odds coefficient = 2.02) at the point where the quadratic term is zero (which is at the 
sample average). The quadratic term is positive, so the relationship is larger among students with 
the highest levels of achievement, and lower among students with low HSGPAs. The school 
variance component for the intercept from this model (0.603) is slightly higher than those in the 
binned model (where variance components ranged from 0.501 to 0.575), and represents the 
variance in school effects averaged across students of all achievement levels. Not only is the 


school-level variance component large (0.603), it is larger when HSGPAs are included in the 
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model than in a model that only includes control variables (0.447, not shown in table). This 
pattern is consistent with the “frogpond” effects discussed earlier, wherein HSGPAs are 
suppressed at high schools with more positive school effects. About one-fourth of the school- 
level variation in Model 2 ((0.603-0.447)/0.603 = 26%) is “extra” variation that is induced by 
comparing students with similar HSGPAs. 

The model displayed in Table 2 also allows the slope of the relationship between HSGPA 
and college graduation to vary by high school. The strong linear trend (coefficient of 0.703), 
does not vary significantly by high school. The quadratic term (coefficient of 0.062) does vary 
slightly across schools (0.103). The noise that is introduced by variation in the linear and 
quadratic components is small relative to the signal from the linear slope (0.703), so the overall 
slope of the relationship is fairly similar across schools. The gray lines in the left panel of Figure 
2 show the relationship of HSGPA with college graduation for each high school, estimated from 
the coefficients and variance components from Model 2. The considerable variation in college 
graduation rates by high school for students with the same HSGPA is clearly visible. At the same 
time, the relationship between HSGPA and college graduation has a similar slope, and is large 
and positive, across high schools. 

Table 3 shows the results of models that mirror those in Table 2, substituting ACT scores 
for HSGPAs. Differences in college graduation rates by ACT score are more modest than by 
HSGPA, particularly after controlling for student background and college characteristics, but 
show a sizable range—from odds of 0.39 to 1.98 in the conditional model (graduation rates of 28 
to 66 percent). School-level variance is smaller among students with the same ACT score than 
among students with the same HSGPA. Still, there is considerable variation in college graduation 


rates by high school among students with the same ACT score (0.265 to 0.343). For students 
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with an ACT score of 16-17, for example, a two-standard deviation range in the log-odds of 
graduating is -0.387 + 0.343. Students with an ACT score of 16-17 in a school with large 
positive effects (one standard deviation above the mean) would graduate at a rate that similar to 
students with scores of 20-21 in a more typical school. Thus, students with the same 
qualifications, defined by either their HSGPA or their ACT score, graduate at different rates 
based upon which high school they attend. 

Model 2 in Table 3 shows the relationship of ACT scores with college graduation 
modeled with continuous linear and quadratic terms. The standardized linear term is much 
smaller than that of standardized HSGPA scores (0.129 vs. 0.703), with the odds of graduating 
increasing by 14 percent (odds coefficient of 1.14) for every standard deviation increase in ACT 
scores when the quadratic term equals zero. There is a negative quadratic term, so the 
relationship is larger among students with low achievement, small among students with high 
achievement, and becomes negative among students with the highest achievement. The variance 
components show that the linear component of the slope varies significantly, and the variance in 
the slopes (0.192) is larger than the average slope (0.129). Thus, the noise introduced by school 
effects is larger than the signal from ACT scores. Where students attend high school says more 
about whether they are likely to graduate from college than their individual ACT score, at least 
among students with average or high ACT scores. 

ACT scores also provide less accurate predictions of college success based on students’ 
race, ethnicity, and gender than HSGPAs. The subgroup differences in college graduation rates 
are significantly different from zero for Asian and male students in the models that control for 
ACT scores, but the demographic coefficients are not significantly different from zero in the 


models that control for HSGPAs. ACT scores explain only a little of the school-level variance in 
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college graduation rates; the variance component on average school effects (0.411) is similar to a 
model with the same control variables but no ACT scores (0.446). However, they do not induce 
more school-level variance, as was seen with HSGPAs. 

The right panel of Figure 1 shows the relationships from Models 2 and 3, modeled as 
percentages. The dark line shows the averages from the bins in Model 2, while the gray lines 
show the relationship for each school, calculated from the coefficients and variance components 
in Model 2. The dark line is not at the center of the gray lines because most of the students with 
high ACT scores are concentrated in schools with high average college graduation rates, while 
students with very low ACT scores are concentrated at schools with low average college 
graduation rates. Many schools do not have students with very high ACT scores, and a number 
of other schools do not have students with very low ACT scores, so few of the lines go the full 
range of the horizontal axis. The figure shows how the relationship of students’ individual ACT 
scores with college graduation is small relative to the variation in. 

In Table 4, ACT scores and HSGPAs are included together in the models. The main 
HSGPA coefficient does not change substantially relative to the model without ACT scores in 
Table 2 (0.708 vs 0.703), but the main ACT coefficient shrinks considerably from the model 
without HSGPA (from 0.129 to a nonsignificant -0.016). Because the ACT score contributes 
little to the prediction, there is a similar amount of school-level variance in the combined model 
(0.622) as the model that includes HSGPA alone (0.603, from Table 1). ACT scores used at the 
individual student level do not reduce the variability by high school in predicting who will 
graduate college. The slope of the relationship of ACT scores with college graduation still varies 
significantly based on high school (0.213); in schools a standard deviation below the mean the 


linear slope is negative (-0.016 - .213, or -0.219) and in others it is positive (-0.016 + 0.213 or 
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0.197). 

In the next model, the variables are group-mean centered so that the coefficients show the 
relationship of each variable with college graduation relative to other students in the same 
school. The school-level variance of the intercept in this model is much larger because the 
student variables do not control for differences across schools in student body composition. The 
within-school coefficient for HSGPAs is slightly larger than the coefficient from the earlier 
model, while the ACT score coefficient is small and not significant. The ACT slope varies 
significantly by high school (0.206, p<0.000) while the main linear portion of GPA slope does 
not vary and the quadratic term varies only slightly. 

In final model, we include predictors of school performance level (average ACT scores 
among all students) and school poverty level. School average ACT scores are significantly 
related to college graduation, explaining school-level differences among students with the same 
HSGPAs and individual ACT scores. The odds of graduating college increase by 60 percent for 
every standard deviation increase in school average ACT scores, for students with the same 
HSGPA and ACT score. Average ACT scores in the school reduce the high school variation in 
college graduation rates by 42 percent (0.324 vs. 0.622). The school poverty level is not 
significant in this model, but that is because it is highly correlated with school average ACT 
scores (r=0.70). If entered alone in the model, either variable is a significant predictor with odds 


ratios of 0.70 for school poverty and 1.68 for school average ACT. 


Discussion 


It is commonly believed that HSGPAs indicate different levels of readiness for college, 


based on the high school a student attended, while ACT scores are consistent indicators. 
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However, HSGPAs perform in a strong and consistent way across high schools as measures of 
college readiness, while ACT scores do not. There are large high school effects on college 
graduation, such that students with either the same HSGPA or the same ACT score graduate 
from college at different rates, based on which high school they attended. Neither capture all of 
the ways in which high schools influence college graduation. The school differences are larger 
for students with the same HSGPA, which is consistent with prior studies showing that grades 
are depressed in schools and classes with higher-achieving students. HSGPAs are not equivalent 
measures of readiness across high schools, but they are strongly predictive in all schools, and the 
signal they provide is larger than the differences across schools. School-level variance in college 
graduation rates is one-quarter smaller among students with the same ACT score than students 
with the same HSGPA. However, this still leaves considerable school-level variance, and the 
signal provided by ACT scores is much smaller than the noise introduced by school effects. 

As measures of individual students’ academic readiness, ACT scores show weak 
relationships, and even negative relationships at the higher achievement levels. The negative 
slope among students with the highest achievement could result if people are using ACT scores 
to make decisions about students’ readiness for very rigorous academic programs out of a belief 
that they are strong indicators of readiness, when they are not. Future research might investigate 
this further. Regardless, there is little evidence that students will have more college success if 
they work to improve their ACT score, as most of the signal from the ACT score seems to 
represent factors associated with the student’s school, rather than the student. In contrast, 
students’ efforts to improve their HSGPAs would seem to have considerable potential leverage 
for improving college readiness. The fact that HSGPAs are based on so many different criteria— 


including effort over an entire semester in many different types of classes, demonstration of 
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skills through multiple formats, and different teacher expectations—does not seem to be a 
weakness. Instead, it might help to make HSGPAs strong indicators of readiness, since they 
measure a very wide variety of the skills and behaviors that are needed for success in college, 
where students will also encounter widely varying content and expectations. 

Test scores provide more of a signal at the school level, with school-level average test 
scores providing additional information about students’ likelihood of graduating above and 
beyond students’ individual HSGPAs. For judging college readiness (e.g., college admissions), 
school-average ACT scores would provide a stronger prediction than students’ individual scores. 
This is consistent with the findings and recommendations in Koretz and Langi (2018) and 
Bowen, Chingos, and McPherson (2009). The same pattern is observed with school-average 
poverty levels (in models that do not control for average ACT scores), which echoes Rothstein’s 
(2004) findings. High school effects could result from higher academic standards (e.g., more 
college-oriented curricula at higher-achieving, higher-SES schools). Yet, they could also 
represent selection effects. Families with more financial, social, and human capital might select 
into higher-achieving, higher-SES high schools, either by choice of residence or application, and 
those families would likely continue to offer financial support when students are in college. 
School effects also could come from different peer networks, advising, supplemental 
experiences, or broader curricular offerings available at schools with more resources. Future 
research should investigate high school effects on college outcomes more thoroughly. 

This study was conducted only with data from Chicago, and only with data from public 
schools. There could be more variation across high schools with a more comprehensive sample, 
and different relationships. The similarity in results that are available from studies of schools in 


other places provide some indication of their generalizability. Studies that use data from samples 
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that include 21 prestigious flagship universities from across the country and all public 
universities in four states (Bowen et al., 2008; Koretz & Langi, 2018; Rothstein, 2004) all show 
that HSGPAs are strongly related to either college graduation or to college freshman GPA, and 
that students’ individual ACT or SAT scores add only modestly to the prediction beyond 
HSGPA, if at all, in models that include high school fixed-effects. The graduation rates presented 
by Bowen, Chingos, and McPherson (2008) for specific HSGPAs are also similar to the 
graduation rates found here and shown in Figure 1. Graduation rates by HSGPA are not provided 
in other studies, to our knowledge. 

This research strongly supports the use of students’ grades in a formative way, to guide 
school improvement efforts and assess the effectiveness of programs designed to improve 
college readiness, and relying much less heavily on test scores. The teachers and schools that 
improve test scores are not always the same as those that improve students’ grades (Jackson, 
2016), and programs that have positive effects on test scores do not always have positive effects 
on grades (Nomi & Allensworth, 2009). Reaching goals that all students will graduate college- 
ready would seem to require strategies around improving students’ HSGPAs, since HSGPAs are 
so strongly related to eventual college completion at all high schools. Higher ACT scores might 
help students get access to stronger colleges, but the pay-off would only occur if students 
actually attend stronger colleges. As an increasing number of colleges become test-optional, they 
are likely to be decreasingly salient for college admissions, as well. 

States and districts might also consider relying less heavily on standardized test scores in 
their accountability systems as indicators of college readiness, given that the relationship is not 
strong and not consistent across schools. A number of states have developed longitudinal data 


systems that allow for the creation of metrics of students’ actual performance in college. The 
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existence of large school effects among students with the same ACT scores suggests that if high 
schools are not tracking the success of their students in college, and are relying solely on 
students’ test scores as indicators of their students’ college readiness, they may be misestimating 
the effects of their practices on students’ college readiness. Likewise, we worry that if families 
and college admissions officers must rely on school poverty levels and average test scores as 
proxy indicators for school effects, they might not recognize strong practices at schools serving 
low-income students. Measuring and publishing school effects on postsecondary outcomes 


would provide better information to guide families, educators, and policymakers. 
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Tables and Figures 


Table 1. Descriptive Statistics of Variables Used in the Models 


Mean Std. Devs 
Gate 


Neighborhood Poverty (standardized 
across all students, not just college- 
goers) 


High School Cumulative HSGPA Pa 2 0.65 
pehicvement ACT Composite Score 20.12 4.33 
College Bi ee 
College Degree in Six Years 49% 

College College Size (# Freshmen) 3662 2390 


Swe % Full Time Freshmen 65% 17% 


Characteristics 
Student to Faculty Ratio 17 5.43 


Six-Year Institutional Graduation Rate 

for student’s racial or ethnic group 47% 22% 
Based on students who enrolled in a four-year college the fall after graduation (n=17,753). 
Institutional characteristics are based on the college freshmen cohort of 2008. 
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Table 2. Model Predicting Six-Year College Graduation Rates by Student HSGPA Score 
Students Nested within High School 


Unconditional Model 1 Model 2 
Coefficients GPA Binned GPA Binned Random GPA slope 

Coeff s.e. odds | Coeff se. odds | Coeff se. odds 
Male -0.096 0.04 0.91 | -0.092 0.04 0.91 
Black -0.024 0.07 0.98 | 0.127 0.08 1.14 
Latino -0.077 0.06 0.93} 0.013 0.07 1.01 
Asian 0.052 0.08 1.05} 0.046 0.08 1.05 
ZPoverty -0.093 0.02 0.91 | -0.069 0.02 0.93 
ZCollege Size 0.061 0.02 1.06; 0.016 0.00 1.02 
Z%Full Time Students 0.013 0.03 1.01} 0.000 0.00 1.00 
ZStudent-Faculty Ratio 0.151 0.03 1.16} 0.833 0.15 2.30 
ZCollege Grad Rate 0.487 0.03 1.62} 0.019 0.00 1.02 
GPA <1.5 -1.834 0.11 0.16 -1.410 0.11 0.24 
GPA 1.5-1.75 -1.720 0.11 0.18 -1.319 0.11 0.27 
GPA 1.75-2.0 -1.210 0.07 0.30 -0.875 0.07 0.42 
GPA 2.0-2.25 -1.429 0.10 0.24 -1.025 0.09 0.36 
GPA 2.25-2.5 -1.185 0.10 0.31 -0.844 0.09 0.43 
GPA 2.5-2.75 -0.834 0.09 0.43 -0.593 0.08 0.55 
GPA 2.75-3.0 -0.496 0.09 0.61 -0.326 0.08 0.72 
GPA 3.0-3.25 -0.098 0.10 0.91 -0.002 0.09 1.00 
GPA 3.25-3.5 0.144 0.10 1.15 0.144 0.08 1.15 
GPA 3.5-3.75 1.296 0.07 3.65 0.939 0.07 2.56 
GPA 3.75-4.0 1.830 0.11 6.23 1.320 0.11 3.74 
ZGPA 0.703 0.03 2.02 
ZGPA2 0.062 0.02 1.06 
Intercept -0.558 0.07 0.57 

Variance of Coefficients across High Schools 
Variance Components In Standard Deviations 
s.d. p-value s.d. p-value s.d. p-value 

GPA 2.25-2.5 0.792 0.000 *** 0.522 0.000 *** 
GPA 2.5-2.75 0.745 0.000 *** 0.598 0.000 *** 
GPA 2.75-3.0 0.754 0.000 *** 0.500 0.000 *** 
GPA 3.0-3.25 0.863 0.000 *** 0.501 0.000 *** 
GPA 3.25-3.5 0.814 0.000 *** 0.575 0.000 *** 
ZGPA 0.107 0.106 
ZGPA? 0.103 0.032 * 
Intercept 0.603 0.000 *** 


*p<0.05, **p<0.01, ***p<0.001. Student background and college institutional control variables were 


grand-mean centered in all models. Variables beginning with Z were standardized, except squared 
terms which are the square of the standardized variables. A model with only the control variables, 
without HSGPA, produces a school-level variance component of 0.447 in standard deviation units. 
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Table 3. Model Predicting 6- Year College Graduation Rates by Student ACT Score 
Students Nested within High School 


Unconditional Model 1 Model 2 
Coefficients ACT Binned ACT Binned Random ACT slope 
Coeff s.e. odds| Coeff s.e. odds | Coeff s.e. odds 
Male -0.346 0.04 0.71 | -0.342 0.04 0.71 
Black -0.045 0.08 0.96 | 0.026 0.08 1.03 
Latino -0.056 0.06 0.95 | -0.012 0.06 0.99 
Asian 0.231 0.10 1.26 | 0.227 0.08 1.25 
ZPoverty -0.091 0.02 0.91 | -0.069 0.02 0.93 
ZCollege Size 0.005 0.02 1.00) 0.009 0.02 1.01 
Z%Full Time Students 0.059 0.02 1.06) 0.058 0.02 1.06 
ZStudent-Faculty Ratio 0.144 0.03 1.15 | 0.139 0.02 1.15 
ZCollege Grad Rate 0.673 0.04 1.96 | 0.671 0.03 1.96 
ACT <14 -1.59 0.11 0.20 | -0.941 0.11 0.39 
ACT14-15 -1.01 0.07 0.37 | -0.482 0.07 0.62 
ACT16-17 -0.793 0.06 0.45 | -0.387 0.06 0.68 
ACT18-19 -0.489 0.06 0.61 | -0.231 0.05 0.79 
ACT20-21 -0.012 0.07 0.99 | -0.059 0.07 0.94 
ACT22-23 0.552 0.09 1.74 | 0.309 0.09 1.36 
ACT24-25 0.852 0.08 2.34] 0.407 0.08 1.50 
ACT26-27 0.986 0.11 2.68 | 0.356 0.10 1.43 
ACT28-29 1.46 0.15 4.33 | 0.684 0.15 1.98 
ACT30+ 1.58 0.17 4.86 | 0.506 0.18 1.66 
ZACT 0.129 0.04 1.14 
ZACT* -0.099 0.02 0.91 
Intercept -0.251 0.06 0.78 
Variance of Coefficients across High Schools 
Variance Components In Standard Deviations 
s.d. p-value s.d. p-value s.d. p-value 

ACT14-15 0.446 .002 ** 0.343 .040* 
ACT16-17 0.447 .000*** 0.343 .000*** 
ACT18-19 0.402 .000*** 0.265 .002** 
ZACT 0.152 012 * 
ZACT? 0.067 .424 
Intercept 0.411 .000 *** 


***4<0.001, **p<0.01, *p<0.05. Student background and college institutional control variables 
were grand-mean centered in all models. Variables beginning with Z were standardized, except 
squared terms which are the square of the standardized variables. A model with only the 
control variables, without ACT scores, produces a school-level variance component of 0.447 in 
standard deviation units. 
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Table 4. Models Predicting 6-Year College Graduation Rates by Both HSGPA and ACT Score 
Students Nested within High School 


Varying Slopes 


Group-Mean Centered 
(School Fixed Effects) 


School-Level Variables 


Male 

Black 

Latino 

Asian 

ZPoverty 

ZCollege Size 

Z%Full Time Students 
ZStudent-Faculty Ratio 
ZCollege Grad Rate 
ZGPA 

ZGPA? 

ZACT 

ZACT? 

ZSchool ave. poverty 
ZSchool Average ACT 
Intercept 

Variance Components 


ZGPA 
ZGPA? 
ZACT 
ZACT? 
Intercept 


Coeff se. odds} Coeff se. odds | Coeff se. odds 
-0.086 0.04 0.92] -0.08 0.04 0.92] -0.08 0.04 0.92 
0.134 0.08 1.14 0.17 0.08 1.19 0.17 0.06 1.18 
0.011 0.07 1.01 0.02 0.07 1.02 0.00 0.07 1.00 
0.042 0.08 1.04 0.03 0.08 1.03 0.04 0.10 1.04 
-0.071 0.02 0.93 -0.06 0.02 0.94] -0.06 0.02 0.94 
0.089 0.02 1.09 0.09 0.02 1.10 0.10 0.02 1.10 
-0.006 0.03 0.99} -0.01 0.03 0.99} -0.01 0.02 0.99 
0.151 0.03 1.16 0.15 0.03 1.16 0.16 0.03 1.17 
0.422 0.04 1.52 0.42 0.04 1.51 0.42 0.04 1.52 
0.708 0.03 2.03 0.73 0.03 2.07 0.75 0.03 2.12 
0.063 0.02 1.06 0.05 0.02 1.05 0.06 0.02 1.06 
-0.016 0.04 0.98] -0.07 0.04 0.94] -0.02 0.04 0.98 
-0.108 0.02 0.90] -0.10 0.02 0.90] -0.08 0.02 0.92 
-0.07 0.05 0.94 
0.47 0.05 1.60 
-0.515 0.08 0.60] -0.70 0.09 0.50] -0.56 0.06 0.57 
Variance of coefficients across high schools 
In Standard Deviations 
0.110 0.246 0.096 0.204 0.112 0.364 
0.106 0.080 0.105 0.036 * Odes OS 7 
0.213 0.002 ** 0.206 .000 peta 0.192 .002 ar 
0.089 >.500 0.088 >.500 0.066 >.500 
0.622 0.000 *** | 0.859 0.000 *** 0.324 0.000 *** 


*p<0.05, **p<0.01, ***p<0.001. Student background and college institutional control variables 
were grand-mean centered in all models. Variables beginning with Z were standardized, except 
squared terms which are the square of the standardized variables. 
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Figure 1. College Graduation Rates by HSGPA and ACT Score, 
Controlling for Student Background and College Characteristics 
Each gray line represents a high school, the black line is the average across high schools 


lessthan 1.5-1.74 1.75-1.99 2.0-2.24 2.25-2.49 2.5-2.74 2.75-2.99 3.0-3.24 3.25-3.49 3.5-3.74 3.75 and Oto 13 14-16 16-17 18-19 20-21 22-23 24-25 26-27 28-29 30+ 


15 higher 
High School GPA 


ACT Score 


Note: Graduation rates by school are calculated from 2-level hierarchical models that allow the relationship between ACT scores or HSGPA to 
vary by high school and include a quadratic term, and control for student race, ethnicity, neighborhood SES, college size, percent full-time 
students, student-faculty ratio, and institutional graduation rate. The average for each point reflects the predicted graduation rate given the average 
HSGPA or ACT score of students in a particular achievement range at each school, which is not always the midpoint. Lines only include HSGPA 
and test score ranges that are observed at the high school, among their college enrollees. The overall rate is calculated from a non-parametric 
model in which HSGPA or ACT scores are entered as a series of dummy variables, along with the same control variables. 
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Appendix 
Table Al. College Graduation Rates by HSGPA and ACT Score 


Overall 


HS GPA 
<1.5 
1.5-1.74 
1.75-1.99 
2.0-2.24 
2.25-2.49 
2.5-2.74 
2.75-2.99 
3.0-3.24 
3.25-3.49 
3.5-3.74 
3.75 and higher 
Overall 


17% 601} 27% 


Grad 
14% 
15% 
23% 
28% 
36% 
44% 
53% 
64% 
68% 
79% 
86% 
49% 


n 
697 
685 
1093 
1708 
2128 
2399 
2497 
2469 
1950 
1382 
745 
17753 


Graduation rates for cells with less than 10 students are not displayed to protect confidentiality. 
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